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Abstract 


High-fidelity  e-learning  preserves  the  quality  of  the  educational  experience  while  minimizing  or 
eliminating  the  need  to  be  in  a  specific  place  at  a  specific  time  with  a  live  instructor.  The  prin¬ 
ciples  of  this  type  of  learning  include  personalization,  multimodality,  lean-forward  learning,  col¬ 
laboration,  accessibility,  modularity,  progress  tracking,  and  balancing  control.  The  SETs  Virtual 
Training  Environment  (VTE)  is  an  example  of  a  high-fidelity  e-learning  success.  The  VTE  was 
developed  to  draw  upon  the  principles  of  high-fidelity  e-learning  to  provide  high-quality  learning 
with  infinite  scalability.  The  purpose  of  this  document  is  to  describe  the  tenets  of  high-fidelity  e- 
learning,  to  describe  how  VTE  reflects  these,  and  to  summarize  how  organizations  have  used  and 
are  using  VTE. 
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Executive  Summary 


This  technical  report  explores  high-fidelity  e-leaming,  which  we  define  as  a  set  of  tools  and  me¬ 
thods  that  delivers  education  and  training  to  students  anywhere ,  anytime ,  without  sacrificing  effi¬ 
cacy.  High-fidelity  e-learning  preserves  the  quality  of  the  educational  experience  while  minimiz¬ 
ing  or  eliminating  the  need  to  be  in  a  specific  place  at  a  specific  time  with  a  live  instructor. 

The  SEI’s  Virtual  Training  Environment  described  in  this  report  has  been  developed  to  reflect  the 
principles  of  high-fidelity  e-leaming  for  workforce  development.  It  offers  the  user  a  web-based, 
multimedia,  interactive  training  experience  that  can  be  accessed  anytime.  This  environment  uses 
state-of-the-art  software  and  hardware  to  train  users  with  technical  and  managerial  roles  on  a  wide 
range  of  information  assurance  topics. 

The  intended  audience  for  this  report  is  responsible  for  building  or  buying  e-learning  programs  for 
employees,  customers,  business  partners,  or  suppliers.  This  report  aids  in  evaluating  alternatives 
and  selecting  solutions  that  need  to  scale  and  change  to  meet  organizational  awareness,  training, 
and  educational  objectives.  The  report  first  describes  the  learning  landscape,  which  ranges  from 
instructor-led  classroom  training  in  a  single  physical  location  to  “pure”  e-learning  where  content 
is  available  anytime,  anywhere,  using  the  internet. 

High-fidelity  e-learning  environments  incorporate  rich  media  technologies,  immersive  simula¬ 
tions,  scenario-based  instruction,  and  hands-on  exercises  that  can  be  accessed  by  students  on  their 
terms  and  as  their  time  permits.  Drawing  from  extensive  research,  the  report  summarizes  the  prin¬ 
ciples  of  high-fidelity  e-leaming  which  include  the  following: 

•  Personalization:  using  a  casual,  conversational  style  for  spoken  and  written  content 

•  Multimodality:  delivering  training  content  using  multiple  sensory  channels  (read  it,  hear  it,  see 
it) 

•  Lean-Forward  Learning:  using  demonstrations,  hands-on  exercises,  and  self  assessments  to 
increase  retention  and  recall  (do  it,  master  it) 

•  Collaboration:  providing  technologies  that  allow  students  to  engage  with  one  another  and  with 
instructors 

•  Accessibility:  making  training  content  broadly  available  to  people  with  a  range  of  abilities 

•  Modularity:  publishing  content  in  manageable,  self-contained  chunks  so  that  it  can  be  easily 
reused  by  content  owners  and  referenced  by  students 

•  Progress  Tracking:  supporting  course  providers,  instructors,  and  students  with  the  ability  to 
track  student  progress 

•  Balancing  Control:  balancing  control  between  the  e-leaming  system  and  the  student  to  achieve 
optimal  learning 
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In  large  part,  these  high-fidelity  e-leaming  principles  are  reflected  in  the  SETs  Virtual  Training 
Environment.  VTE  (http://vte.cert.org)  was  launched  in  2004  to  improve  the  scalability  of  tech¬ 
nical  training.  VTE  holds  over  1,800  unique  modules,  representing  740  hours  of  training,  and  cur¬ 
rently  delivers  over  10,000  hours  of  training  a  month  in  support  of  U  .S.  DoD  and  civilian  gov¬ 
ernment  agencies  and  the  general  public. 


C  ours  e  material  on  V  TE  i  s  structure  d  to  all  o w  stud  ents  to  pro  gre  ss  at  their  own  pa  ce,  using  a 
Read  It,  Hear  It,  See  It,  D  o  It,  Master  It  approach: 


READ 

rr 

T opics  begin  with  background  do  cumentation,  related  policy,  and  oth¬ 
er  foundational  material. 

HEAR 

rr 

A  video  camera  placed  in  a  student  seat  records  the  entire  class,  which 
is  then  published  to  the  web  in  1 5 -minute  blocks  suitable  for  on- 
demand  viewing. 

see  rr 

Narrated  ‘screencast’  walkthroughs  of  specific  applications  and  best 
practices  reinforce  the  lecture  material. 

m 

DO  IT 

Hands-on  labs  provide  an  active  learning  experience.  Students  can  re¬ 
motely  stand  up  any  of  VTE' s  70  computer  network  configurations  in 
under  three  minutes.  This  allows  them  to  learn  by  doing  and  to  practice 
on  software  that  could  not  be  safely  loaded  on  their  own  machines. 

4* 

MASTER 

rr 

All  material  is  available  on  demand,  so  technical  training  isn’t  some¬ 
thing  students  attend;  it  is  something  they  always  have.  Students  can 
rep  eat  any  portionofthecourse  material  days,  we  eks,  months ,  or  years 
after  taking  the  class. 

Recreating  the  classroom  instruction  experience,  providing  a  verbatim  text  transcript,  presenting 
demonstrations  and  opportunities  to  engage  hands-on  labs  at  critical  learning  points,  providing  for 
progress  tracking  and  self-assessment,  and  offering  virtual  office  hours  are  a  few  examples  of  how 
VTE  has  incorporated  high  fidelity  e-leaming  principles. 

The  report  closes  with  experience  reports  from  two  U.S.  government  organizations  that  used  VTE 
to  help  meet  their  training  requirements.  Appendix  A  describes  the  experiences  of  D  ISA  in  ad¬ 
dressing  U.  S.  Department  of  Defense  Directive  8570.1  requirements  for  infonnati  on  assurance 
workforce  improvement.  Performance  and  cost  data  from  the  DISA  contract  are  compared  to  an¬ 
nual  survey  data  from  the  American  Society  of  Training  and  Development's  “2007  State  of  the 
Industry  Report.”  VTE  was  used  to  deliver  38,157  hours  of  training  for  DISA  during  the  period 
from  1/1/2007  through  10/31/2007.  Based  ontheASTD  comparison,  DISA  saved  more  than 
$  1 . 2M  with  a  return-  on-investment  o  f  1 4 1  %  when  comparing  V  TE  co  sts  to  what  it  c  oul  d  have 
expected  to  pay  at  prevailing  industty  average  costs. 
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Appendix  B  summarizes  the  U.  S.  Secret  Service  Criminal  Investigative  Division’s  experiences 
using  VTE  to  help  train  its  computer  investigative  specialists  in  digital  examination  using  distance 
learning  technologies.  This  study  took  place  from  October  1,  2005  through  November  30,  2005. 
“Overall,  participants  rated  VTE’s  features  and  functionality  between  seven  and  ten  on  a  10-point 
scale.  Areas  rated  included  access,  navigation,  and  audio  and  video  content.” 

Organizations  that  wish  to  discuss  leveraging  high-fidelity  e-learning  in  their  environment  should 
contact  VTE  Support  at  vte-support@cert.org  to  discuss  training  initiatives  and  next  steps. 
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1  Introduction 


Since  its  inception  in  1984,  the  Software  Engineering  Institute  (SEI)  has  been  leveraging  distance 
education.  Initially,  the  SEI  recorded  lectures  from  its  software  engineering  curriculum  to  video¬ 
tape  and  mailed  them  to  remote  students  and  offices.  As  standards  for  video  on  computers 
emerged,  the  SEI  shifted  to  distributing  CD  and  DVD-ROM  versions  of  recorded  lectures.  As  the 
SEI’s  distance  and  e-learning  offerings  evolved,  best  practices  began  to  emerge.  Each  iteration  of 
the  e-learning  environment  built  upon  the  lessons  learned  from  organizations  and  students  using 
the  environment. 

This  introductory  section  describes  the  intended  audience  and  desired  outcomes  for  this  technical 
report.  It  then  contrasts  e-leaming  with  other  modes  of  instruction  and  sets  the  stage  for  describ¬ 
ing  the  principles  of  high-fidelity  e-learning  (Section  2)  and  how  they  are  reflected  in  the  SEI's 
Virtual  Training  Environment  (VTE)  (Section  3).  These  principles  can  serve  as  criteria  for  eva¬ 
luating  candidate  e-learning  environments  that  are  being  considered  to  satisfy  organizational  edu¬ 
cation  and  training  requirements.  Section  3  includes  a  description  of  the  VTE  architecture  and 
data  on  VTE  use  to  date. 

The  final  section  of  the  report  presents  two  perspectives  for  using  VTE:  that  of  the  producer  or 
author  of  new  VTE  content  (including  lectures,  demonstrations,  and  hands-on  labs)  and  that  of  a 
typical  user  pursuing  an  information  security  certification.  Appendix  A  describes  the  U.S.  De¬ 
partment  of  Defense's  (DoD)  and  DISA’s  experiences  using  VTE  to  meet  U.S.  DoD  directive 
requirements  for  information  assurance  workforce  competence  as  well  as  a  return-on- investment 
analysis  compared  to  benchmark  data  from  the  American  Society  of  Training  and  Development. 
Appendix  B  contains  an  excerpt  from  a  VTE  pilot  study  conducted  by  the  U.  S.  Secret  Service. 

1.1  Target  Audience 

The  audience  for  this  report  is  any  individual  or  member  of  an  organization  charged  with  acquir¬ 
ing,  developing,  or  deploying  e-learning  programs  within  and  across  organizational  boundaries. 

1.2  Desired  Outcomes 

After  reviewing  this  report,  readers  will  better  understand  the  concept  of  high-fidelity  e-leaming 
as  it  relates  to  organizational  training  initiatives,  and  will  be  familiar  with  the  SEI’s  Virtual  Train¬ 
ing  Environment  as  an  implementation  of  a  high-fidelity  e-leaming  system.  This  report  does  not 
cover  concepts  of  pedagogy  or  instructional  design  in  e-leaming  or  any  other  medium,  except  as 
they  influence  the  delivery  model  for  training.  That  said,  we  do  recognize  that  poor-quality  train¬ 
ing  material  can  undermine  a  student’s  ability  to  meet  objectives  regardless  of  the  delivery  envi¬ 
ronment. 

1.3  e-Learning,  Classroom  Learning,  and  Blended  Learning 

Learning  models  are  frequently  described  as  points  on  a  spectrum.  As  depicted  in  Figure  1,  at  one 
extreme  is  instructor-led  training  (ILT,  also  referred  to  as  classroom  training  or  direct  instruction). 
At  the  other  end  of  the  spectrum  is  a  family  of  techno  logy -based  models  including  computer- 
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based  training  (CBT)  and  web-based  training  (WBT).  Collectively,  models  at  this  end  are  grouped 
into  the  e-learning  category.  Blended  learning  occurs  between  the  two  extremes,  which  is  a  com¬ 
bination  of  direct  instruction  and  e-leaming. 


Figure  1:  Spectrum  of  Learning/Training  Models 

While  classroom  instruction  as  an  educational  method  is  fairly  well  understood,  definitions  for 
terms  such  as  e-leaming  and  blended  learning  are  still  the  subject  of  debate.  In  this  report,  we  at¬ 
tempt  to  avoid  the  philosophical  discussion  inherent  in  these  terms  and  instead  map  training  mod¬ 
els  based  on  their  logistical  requirements.  Different  types  of  training  methods  may  require  that 
teacher  and  student  be  in  the  same  physical  location  (physical  vs.  virtual)  and/or  require  they  meet 
at  the  same  time  (synchronous  vs.  asynchronous).  Traditional  classroom  instruction  assumes  the 
instructor  and  student  are  in  the  same  location  at  the  same  time.  Virtual  classrooms  use  technolo¬ 
gy  to  extend  the  synchronous  classroom  experience  to  any  location.  CBT,  training  libraries,  and 
other  non-networked  training  materials  are  available  only  in  specific  locations  but  can  be  used  at 
any  time.  ‘ Pure  e-leaming  assumes  that  self-paced  materials  are  readily  available.  Blended  learn¬ 
ing  combines  several  of  these  approaches,  using  asynchronous  and  virtual  training  methods  with 
classroom  instruction  as  the  anchor.  As  an  expansion  to  the  spectrum  presented  in  Figure  1 ,  Fig¬ 
ure  2  plots  the  different  training  models  on  a  grid  (note  that  the  grid  excludes  the  categories  of 
non-computer  technology  and  other). 
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Figure  2:  Training  Models  Shown  in  Two  Dimensions 
Dimensions 

According  to  the  American  Society  of  Training  and  Development  (ASTD)  2007  member  survey 
[ASTD  2007],  self-paced  asynchronous  e-leaming  is  the  fastest  growing  training  model,  coming 
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primarily  at  the  expense  of  classroom  instruction.  The  percentages  in  the  grid  represent  the  por¬ 
tions  of  the  overall  professional  training  market  served  by  each  type. 

Another  factor  that  varies  by  type  of  instruction  is  the  cost  to  provide  the  education.  Cost  in  this 
case  is  a  combination  of  direct  costs  for  facilities,  technology,  and  transportation  as  well  as  oppor¬ 
tunity  costs  for  instructors  and  students.  The  logistical  requirements  of  synchronous  training  in  a 
specific  location  make  it  the  most  costly  method.  Synchronous  virtual  training  and  asynchronous 
physical  training  each  offer  some  cost  savings  over  classroom  instruction,  but  pure  e-learning’s 
anywhere,  anytime  availability  gives  it  a  decided  cost  advantage  over  all  other  forms.  Additional 
information  on  the  cost  advantages  of  e-learning  is  provided  in  Appendix  A. 

1 .4  High-Fidelity  E-Learning 

This  report  focuses  on  pure  e-leaming.  Though  we  recognize  some  situations  work  best  in  a  syn¬ 
chronous  environment,  given  the  cost  and  logistical  savings  associated  with  asynchronous,  self- 
paced  e-learning,  this  report  focuses  on  this  method  of  instruction.  Furthermore,  this  report  ele¬ 
vates  the  concept  of  e-leaming  to  the  distinction  high-fidelity  e-learning. 

We  define  high-fidelity  e-leaming  as  a  set  of  tools  and  methods  that  delivers  education  and  train¬ 
ing  to  students  anywhere ,  anytime  without  sacrificing  ejficacy.  High-fidelity  e-leaming  preserves 
educational  quality  while  minimizing  or  eliminating  logistical  requirements.  High-fidelity  e- 
learning  begins  with  the  assumption  that  instructional  delivery  will  be  primarily  asynchronous,  so 
the  content  and  its  presentation  must  minimize  the  requirement  for  a  live  instructor. 

The  next  section  describes  the  foundational  principles  that  characterize  a  high  fidelity  e-leaming 
environment. 


3  |  CMU/SEI-TR-2009-005 


2  Principles  of  High  Fidelity  e-Learning 


High-fidelity  e-learning  extends  the  traditional  concepts  of  e-learning  through  the  use  of  rich 
media  technologies,  immersive  simulations,  and  scenario-based  instruction  while  maintaining  its 
anywhere,  anytime  foundation.  It  is  based  on  the  presence  of  a  ubiquitous  internet  connection, 
representing  an  always-connected  world.  It  assumes  students  can  and  will  access  material  on  their 
terms  and  that  they  will  acquire  and  maintain  skills  while  they  fulfill  their  day-to-day  responsibili¬ 
ties.  High-fidelity  e-learning  supports  and  encourages  hands-on  learning,  allowing  students  to 
practice  instructional  concepts  whenever  and  wherever  they  choose. 

This  section  describes  the  eight  foundational  principles  that  characterize  a  high  fidelity  e-learning 
environment  and  describe  the  user  experience. 

2.1  Personalization 

As  summarized  by  Clark  and  Mayer,  research  has  shown  that  e-learning  is  enhanced  when  a  con¬ 
versational  style  is  used  for  both  spoken  and  written  text.  It  also  benefits  from  the  use  of  on¬ 
screen  pedagogical  agents.  This  is  the  basis  for  Clark  and  Mayer’s  Personalization  principle  for 
designing  media  for  e-learning  [Clark  Sc  Mayer  2003]. 

E-learning  designers  and  developers  may  be  tempted  to  incorporate  precise  and  formal  text  on¬ 
screen  and  in  narration  read  by  a  professional  voice  talent.  The  personalization  principle  counters 
this  temptation  and  reminds  us  that  a  casual,  personal  style  of  communication  is  more  effective 
for  learning.  The  challenge  for  e-learning  is  to  replicate  the  effective  and  personal  communica¬ 
tions  that  often  happen  in  engaging  classroom  environments. 

Current  technologies  make  video  segments  of  classroom  instruction  easy  to  integrate  with  e- 
learning  courseware  and  therefore  make  the  experience  of  the  classroom  and  the  instructor  availa¬ 
ble  not  only  to  present  material,  but  also  to  coach,  work  through  examples,  and  guide  the  learning 
process  in  other  ways.  The  use  of  video  streaming  in  e-learning  can  make  the  classroom  instructor 
available  as  the  pedagogical  agent. 

2.2  Multimodality 

Multimodality  is  defined  as  “the  use  of  two  or  more  of  the  five  senses  for  the  exchange  of 
information”  [Granstrom,  House,  Sc  Karlsson  2002].  Multimodality  is  a  broad,  multidisciplinary 
area  of  research  that  includes  the  study  of  human-to-human  spoken  and  gestural  communications, 
audio-visual  speech  perception,  and  the  me oq:> oration  of  multiple  modalities  into  human-to- 
system  communication.  As  Granstrom  et  al  point  out,  multimodality  is  not  new,  “From  the  early 
storytelling  tradition,  where  body  and  facial  gestures  have  always  played  a  major  role,  through 
classic  theater,  opera,  dance,  film,  video  and  multimedia,  multimodal  communication  has  been 
constantly  present  in  human  communication  and  human  culture.”  The  multimodality  principle  for 
high-fidelity  e-leaming  is  to  deliver  content  across  multiple  sensory  channels  (visual  and  audible) 
to  aid  learning. 
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Clark  and  Mayer  advocate  the  principles  of  multimedia  and  modality  as  guidelines  for  the  design 
of  e-learning.  Their  multimedia  principle  calls  for  including  both  words  (written  or  spoken)  and 
graphics  in  e- learning  materials  [Clark  &  Mayer  2003],  Res  each  and  cognitive  learning  theory 
indicate  that  people  learn  more  deeply  from  words  and  pictures  than  from  words  alone.  The 
modality  principle  recommends  presenting  words  as  speech  rather  than  onscreen  text.  According 
to  cognitive  learning  theory,  people  have  different  information  processing  channels  for  visual 
information  and  for  audio  information;  presenting  information  simultaneously  across  the  two 
channels  reduces  the  likelihood  that  either  channel  is  overloaded  and  therefore  aids  learning. 

Multimodality  can  also  be  described  as  read  it,  hear  it,  see  it  learning.  Classrooms  offer  rich  op¬ 
portunities  for  multimodality  such  as  chalk  and  whiteboards,  document  cameras,  handouts,  and 
instructor  speech,  gestures,  facial  expressions,  and  body  language.  In  e-learning,  the  role  of  video 
can  help  to  make  the  richness  of  gestural  and  body  language  communication  available  from  the 
classroom,  but  it  needs  to  be  balanced  with  graphics  and  other  supporting  media  to  enable  the  full 
multimodal  experience. 

An  example  of  how  the  e-leaming  environment  can  promote  multimodality  could  involve  the  fol¬ 
lowing  elements: 

•  Read  it:  transcripts  of  the  instructor’s  presentation  are  available. 

•  Hear  it:  audio  of  the  instructor  maps  to  the  video  of  the  presentation. 

•  See  it:  presentation  materials,  screenshots,  and  diagrams  appear  on-screen  synchronized  with 
the  instructor’s  words. 


2.3  Lean-Forward  Learning 

E-learning  is  a  new  educational  paradigm.  In  part,  e-learning  represents  a  shift  away  from  instruc¬ 
tor-centered  learning  toward  student-centered  learning.  The  web  provides  access  to  vast  amounts 
of  information  anytime  and  anywhere,  which  makes  information  gathering  and  synthesis  an  easy 
part  of  the  process  of  life-long  learning.  E-learning  offers  self-paced  courses  in  a  web  environ¬ 
ment,  allowing  for  the  pursuit  of  more  formal  learning  throughout  our  lifetimes  without  the  re¬ 
quirement  to  be  in  a  particular  place  at  a  set  time. 

As  a  principle  of  high-fidelity  e-learning,  lean-forward  learningrefers  to  the  inclusion  of  hands- 
on,  context-appropriate  practices,  demonstrations,  exercises,  and  assessments  to  engage  the  stu¬ 
dent  and  to  assist  with  encoding  the  material  being  learned  in  a  way  that  facilitates  recall  in  the 
future. 

Lean-forward  learning  includes  two  distinct  e-leaming  activities: 

1 .  practices,  exercises,  and  labs 

2.  formative  assessments 
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2.3.1 


Practices,  Exercises,  and  Labs 


Based  on  their  review  of  learning  research  and  theory,  Clark  and  Mayer  advocate  the  incorpora¬ 
tion  of  practice  exercises  and  examples  into  e-learning  courses.  Practices,  exercises,  and  labs  are 
key  learning  activities  that  provide  the  student  with  the  opportunity  to  use  the  knowledge  being 
acquired  to  complete  context-relevant  problems  or  tasks.  The  practice  exercises  and  examples 
should  mirror  the  job  context  as  closely  as  possible  to  help  encode  knowledge  in  a  manner  that 
supports  its  ultimate  retrieval  when  needed  to  support  task  performance  or  problem  solving  in  the 
workplace.  They  also  report  on  evidence  that  practices  distributed  throughout  the  training  results 
in  better  long-term  retention  than  the  same  practices  placed  at  the  end  of  the  training  activity 
[Clark  Sc  Mayer  2003], 

For  IT-related  content,  virtual  machine  technologies  can  be  used  to  create  learning  laboratories  in 
which  students  can  practice  configuration,  diagnosis,  and  other  problem-solving  skills  in  an  envi¬ 
ronment  that  accurately  simulates  their  on-the-job  experience.  This  should  lead  to  better  retention 
and  recall  of  the  skills  being  trained.  Virtua  l  machine-based  labs  are  also  particularly  well  suited 
for  asynchronous  learning  because  they  can  be  reset  and  allocated  to  a  particular  student  on  de¬ 
mand,  without  instructor  intervention. 

Practices,  exercises,  and  labs  extend  the  read  it,  hear  it,  see  it  model  introduced  in  the  multimo¬ 
dality  principle  to  create  the  read  it,  hear  it,  see  it,  do  it  model  for  high-fidelity  e-learning.  This 
model  provides  students  the  opportunity  to  engage  the  learning  topic  in  a  multimedia  and  multi¬ 
modal  environment  and  provides  both  active  as  well  as  passive  opportunities  for  students  to  leam 
at  their  own  pace.  Being  able  to  replay  VTE  content  is  a  key  advantage  over  a  standard  classroom 
environment. 

2.3.2  Formative  Assessment 

Assessment  has  long  been  a  key  part  of  the  educational  experience.  There  are  two  types  of  learn¬ 
ing  assessments:  summative  assessments  are  used  for  grading  or  ranking  students  and  for  confirm¬ 
ing  certain  student  achievements  (e.g.,  certification  exams);  formative  assessments,  or  assess¬ 
ments  for  learning,  are  used  to  provide  feedback  to  the  student  to  highlight  areas  of  further  study 
and  improve  performance  [McAlpine  Sc  Higgison  2001],  Because  this  report  is  about  self-paced 
e-learning,  we  focus  on  formative  assessments. 

In  studies  of  formative  computer  assisted  assessments  (CAA)  in  the  form  of  multiple  choice  ques¬ 
tions,  students  have  reported  that  such  assessments  are  helpful  in  preparing  for  summative  as¬ 
sessments  [Iahad,  Dafoulas,  Kalaitzakis,  Sc  Macaulay  2004],  useful  for  checking  their  progress  in 
an  e-learning  environment,  and  helpful  for  improving  the  understanding  of  content  [Iahad  Sc 
Dafoulas  2004]. 

“Formative  CAA  can  assist  in  consolidation  of  learning,  and  in  identifying  weaknesses  in  as¬ 
sumed  understanding.  In  addition  to  the  specific  learning  effects  that  immediate  feedback  has 
within  an  online  course,  formative  assessment  may  have  a  more  general  effect  on  students  by  en¬ 
hancing  self-assessment  of  understanding.  This  metacognitive  skill  (of  knowing  when  you  know 
something  and  when  you  don’t)  is  considered  by  many  to  be  a  key  feature  of  tertiary -level  educa¬ 
tion,  and  it  is  surprising  that  the  “lowly”  multiple  choice  question  has  the  potential  to  play  a  role 
in  the  development  of  this  higher-order  cognitive  skill”  [Dalziel  2001], 
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High-fidelity  e-learning  should  offer  students  opportunities  to  gauge  their  understanding  of  the 
material.  These  formative  assessments  do  not  have  to  be  and  shouldn’t  be  punitive  or  serve  as 
gates  on  the  student’s  progress  through  the  material.  They  should  be  a  chance  for  the  student  to 
decide  whether  he  or  she  should  continue  to  review  the  current  module  or  is  comfortable  with  the 
material  and  should  proceed  to  the  next  module. 

Formative  assessments  extend  the  read  it,  hear  it,  see  it,  do  it  model  to  create  the  read  it,  hear  it, 
see  it,  do  it,  master  it  model  for  high-fidelity  e-learning.  Master  it  accurately  describes  the  role  of 
formative  assessments  in  high-fidelity  e-learning,  because  such  assessments  enable  and  empower 
the  student  to  challenge,  test,  and  confirm  their  mastery  of  the  material. 

2.4  Collaboration 

Many  research  studies  have  shown  that  in  conventional  courses,  students  who  work  or  study  to¬ 
gether  learn  more  than  those  who  work  or  study  alone  [Clark  8c  Mayer  2003],  In  a  three-year 
comparative  study  of  online  and  conventional  courses,  Benbunan-Fich  and  Hiltz  found  that  “out¬ 
comes  of  online  courses  improved  when  professors  structured  them  to  support  the  growth  of  a 
learning  community,  by  being  available  online  to  interact  with  students,  and  by  using  collabora¬ 
tive  learning  strategies”  [Benbunan-Fich  8c  Hiltz,  2003]. 

Evidence  is  building  that  collaboration  among  students  and  with  an  instructor  improves  learning 
from  online  courses.  We  address  this  principle  of  collaboration  in  two  parts: 

1 .  peer- student  collaboration 

2.  instructor  support 

2.4.1  Peer-Student  Collaboration 

Research  has  indicated  that  “working  in  groups,  instead  of  alone,  increases  motivation,  perception 
of  skill  development  and  solution  satisfaction”  [Hiltz  8c  Benbunan-Fich  1997],  This  finding  was 
based  on  self-reported  learning,  with  groups  working  online  through  an  asynchronous  learning 
network  reporting  slightly  better  perceptions  of  learning  than  the  groups  working  face-to-face.  In 
another  study,  86  percent  of  students  in  online  courses  reported  that  working  in  teams  was  helpful 
for  learning  [Lee,  Magjuka,  Liu,  8c  Bonk  2006], 

Many  other  research  studies  have  found  that  collaboration  among  students  aids  learning. 
Benbunan-Fich  and  Hiltz  offer  two  arguments  to  explain  why  participating  in  a  group  helps 
people  to  learn:  1)  it  mediates  so cio- emotional  variables  (e.g.,  increases  motivation  and  reduces 
anxiety)  to  create  a  favorable  climate  for  learning,  and  2)  it  provides  mechanisms  that  impact 
cognitive  processes  (e.g.,  resolution  of  disagreements  through  group  discussions,  internalization 
of  explanations  by  others,  and  self- explanation  effects).  (Benbunan-Fich  8c  Hiltz,  2003). 

Two  major  challenges  to  the  effective  use  of  collaboration  among  peer  students  in  high-fidelity  e- 
learning  involve  enabling  collaboration  and  motivating  collaborative  learning  opportunities 
through  appropriate  instructional  design  or  pedagogy. 
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Many  forms  of  computer-aided  collaboration  are  readily  available  to  enable  communication  and 
collaboration  in  e-leaming.  As  summarized  in  Table  1,  they  comprise  both  synchronous  and  asyn¬ 
chronous  technologies. 

Table  1:  Computer-aided  Communication  and  Collaboration  Technologies 


Synchronous 

Asynchronous 

•  Chats 

•  Message  boards 

Online  conferencing 

|>  |  E-mail 

•  VOIP  telephony 

•  Threaded  discussion  boards 

Virtual  worlds 

b  1  Listservs 

•  Multi-player  internet  mediated  gaming 

•  Message  boards 

b  Wikis 

*  Blogs 

fj  Social  networks 

•  Shared  file  spaces 


Although  this  report  focuses  on  high-fidelity  e-learning,  which  we  have  defined  as  an  asynchron¬ 
ous  activity,  we  include  synchronous  collaboration  techniques  here  because  they  can  be  useful  in 
support  of  any  online  course  with  sufficient  concurrency  —  a  measure  of  the  number  of  students 
actively  using  an  online  course  at  any  one  point  in  time  [Clark  8c  Mayer  2003].  High  concurrency 
can  be  experienced  in  asynchronous  courses  with  a  very  large  numbers  of  participants  or  in  aca¬ 
demic,  for-credit  courses  that  are  temporally  bound  by  the  academic  semester  schedule.  In  online 
courses  with  extremely  low  concurrency,  asynchronous  communication  and  collaboration  tech¬ 
niques  may  not  be  feasible  or  useful  because  there  are  not  sufficient  numbers  of  students  engaged 
in  the  course  material  at  any  time. 

It  is  clear  that  peer-student  collaboration  improves  learning  and  should  therefore  be  facilitated  by 
high-fidelity  e-leaming  systems.  However,  the  issue  associated  with  designing  and  motivating 
effective  collaboration  in  online  learning  is  an  instructional  design  issue  and  is  beyond  the  scope 
of  this  paper. 

2.4.2  Instructor  Support 

Whether  called  instructor,  teacher,  mentor,  or  guide,  an  experienced  individual  delivering  or  faci¬ 
litating  access  to  training  material  adds  well-established  value.  Self-paced,  location-independent 
e-learning  should  provide  access  to  an  instructor  when  the  student  requests  it.  Two  main  prin¬ 
ciples  apply: 

•  The  instructor’s  or  subject  matter  expert’s  availability  and  contact  information  should  be  rea¬ 
dily  available  to  students. 

•  The  instructor  should  make  every  attempt  to  minimize  the  time  between  the  student’s  request 
and  the  instructor’s  offer  of  assistance.  The  goal  would  be  to  reduce  the  time  delta  to  zero  - 
the  student’s  question  is  answered  immediately.  However,  because  the  student  can  return  to 
the  material  at  any  time,  the  instructor’s  response  is  still  valuable  to  the  student  even  if  it  is 
delayed. 

Timely  instructor  support  and  interaction  will  be  subject  to  course  concurrency;  without  sufficient 
concurrency,  active  instructor  support  may  not  be  feasible.  Many  of  the  same  computer-mediated 
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technologies  listed  in  Table  1  are  appropriate  for  instructor- student  communication  and  collabora¬ 
tion. 

2.5  Accessibility 

The  accessibility  principle  for  high-fidelity  e-learning  is  to  deliver  content  in  a  manner  that  is 
broadly  accessible  to  people  with  a  wide  range  of  abilities. 

In  the  United  States,  Section  508  of  the  Rehabilitation  Act  “establishes  requirements  for  electron¬ 
ic  and  information  technology  developed,  maintained,  procured,  or  used  by  the  Federal  govern¬ 
ment.  Section  508  requires  Federal  electronic  and  information  technology  to  be  accessible  to 
people  with  disabilities,  including  employees  and  members  of  the  public”  [U.S.  Department  of 
Justice,  Civil  Rights  Division,  Disability  Rights  Section  2005].  Even  though  it  does  not  explicitly 
apply  to  the  training  activities  or  websites  of  private  organizations.  Section  508  has  become  the  de 
facto  standard  for  making  such  material  accessible. 

Two  subsections  of  technical  standards  (subpart  B)  provided  in  Section  508  apply  to  online 
courses  (IT  Accessibility  Sc  Workforce  Division  (ITAW),  Office  of  Governmentwide  Policy,  U.S. 
General  Services  Administration): 

§  1 194.22  Web-based  intranet  and  internet  information  and  applications. 

This  section  applies  to  all  websites  and  is  not  unique  to  training  materials  or  online  courses. 
The  standard  requires  that  web  content  be  constructed  and  tagged  in  a  manner  that  enables 
the  effective  use  of  screen  readers  and  other  technologies  that  allow  people  with  disabilities 
to  access  internet  content.  The  commercial  marketplace  includes  numerous  design  tools, 
checking  methods,  and  training  courses  to  assist  website  owners  to  comply  with  Section  508. 

§  1194.24  Video  and  multimedia  products. 

This  section  contains  two  provisions  that  apply  explicitly  to  training  and  informational  video 
and  multimedia  productions.  These  provisions  require  a)  captioning  of  any  speech  or  other 
audio  information  that  is  necessary  for  the  comprehension  of  the  material  and  b)  audio  de¬ 
scriptions  of  any  visual  information  that  is  necessary  for  the  comprehension  of  the  material. 

Some  accessibility  features  provide  benefits  to  all  users.  For  example,  a  full  transcript  for  audio 
content  provides  all  users  with  a  means  to  better  understand  the  lecture  content.  This  is  particular¬ 
ly  helpful  when  students  are  not  native  speakers  of  the  lecture  language  or  are  new  to  the  technol¬ 
ogy.  Transcripts  could  be  provided  in  multiple  languages  to  support  multi-national  training  pro¬ 
grams. 

Transcripts  are  more  flexible  and  offer  distinct  advantages  over  captions.  Captions  often  obscure 
part  of  the  video  content  and  are  typically  implemented  as  part  of  the  video  stream.  Transcripts 
are  displayed  in  a  separate  window  and  do  not  obscure  the  video  content.  Transcripts  also  offer 
the  opportunity  of  user-override  control.  With  this  feature,  a  student  who  is  familiar  with  certain 
parts  of  the  material  can  scan  forward  on  the  transcript  to  find  the  new  or  unfamiliar  material  and 
have  the  video  stream  re-synchronize  with  that  section  of  the  transcript.  This  is  also  helpful  for 
students  who  want  to  return  to  the  training  module  to  review  or  reference  the  training  material. 


9  |  CMU/SEI-TR-2009-005 


2.6  Modularity 


Modularity  means  that  the  high-fidelity  e-learning  system  should  support  and  loosely  enforce 
constraints  on  content  managed  and  delivered  by  the  system.  These  constraints  may  take  the  form 
of  metadata,  navigation  rules,  and  schemas  that  encourage  content  creators  to  publish  material  as 
individual,  self-contained  blocks  of  instruction.  Implementing  modularity  must  be  balanced 
against  the  constraints  it  may  impose  on  the  design  of  the  learning  material  and  outline.  Modulari¬ 
ty  includes  two  sub -principles: 

1 .  design  for  re-use 

2.  design  for  reference 

2.6.1  Design  for  Re-Use 

Reusability  is  one  area  where  e-learning  offers  natural  advantages  over  direct  instruction.  Once 
developed,  e-learning  content  can  be  transferred  and  replayed  infinitely.  If  properly  designed,  e- 
learning  content  can  be  combined  with  other  e-learning  materials  for  rapid  course  creation  with 
minimal  redevelopment.  E-learning  standards  such  as  SCORM1  and  Common  Cartridge^  mandate 
reusability  in  conformant  e-learning  material.  High-fidelity  e-learning  should  support  reuse  ba¬ 
lanced  against  the  other  principles  described  in  this  section. 

High-fidelity  e-learning  systems  should  also  encourage  content  authors  to  search  existing  material 
before  creating  new  material.  For  individual  modules,  if  existing  content  is  suitable  for  the  in¬ 
structional  needs,  the  content  owner’s  problem  is  solved.  For  course  development,  the  high- 
fidelity  e-learning  system  should  assist  in  developing  the  outline  and  present  candidate  material 
for  inclusion  based  on  the  course  creator’s  searches  and  interests. 

2.6.2  Design  for  Reference 

Learning  involves  both  skill  building  and  skill  maintenance.  Postsecondary  training  often  focuses 
on  specialized  job  activities,  tasks  that  are  performed  infrequently,  or  tasks  that  are  specific  to  an 
organization.  In  these  cases,  it  is  valuable  for  learning  content  to  be  available  as  a  reference  source 
for  future  use  to  support  knowledge  recall  or  skill  refreshment  at  the  time  the  tasks  are  to  be  per¬ 
formed. 

Content  owners  should  not  assume  that  the  course  for  which  a  module  was  developed  is  the  only 
time  or  place  it  will  be  used,  and  the  high-fidelity  e-learning  system  should  offer  mechanisms, 
such  as  indexing  and  embedding,  that  allow  individual  modules  to  be  used  in  ways  the  content 
owners  did  not  originally  envision. 

Making  the  content  easy  for  the  student  to  find  and  review  long  after  a  particular  course  is  com¬ 
plete  supports  recall  and  mastery  of  the  material  —  an  important  element  of  Read  It,  Hear  It,  See 
It,  Dolt,  Master  It  learning. 


Sharable  Content  Object  Reference  Model;  http ://www. adlnet.gov/scorm/ 
http://www.imsglobal.org/commoncartridge.htnnl 
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2.6.2. 1  Indexing 


The  high-fidelity  e -learning  system  should  make  content  indexable  for  public  and  private  search. 
Rich  media  should  include  a  transcript  searchable  from  outside  the  content.  Any  text  element 
should  be  indexed  for  search.  If  the  learning  module  is  based  on  a  timeline,  the  search  indexing 
should  support  linking  directly  to  the  search  term.  High-fidelity  e-learning  should  make  it  easy 
and  natural  to  add  metadata  to  published  content  to  support  searching  across  and  within  modules 
and  filtering  based  on  characteristics  of  the  material.  The  Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  Learning  Object  Metadata  standard3 and  the  Dublin  Core  Metadata  Initiative4 de¬ 
fine  sets  of  metadata  for  learning  objects  and  a  common  data  structure  that  can  be  implemented 
across  disparate  learning  management  systems.  We  recognize,  however,  that  these  standards  can 
be  overwhelming  to  a  content  publisher.  High-fidelity  e-leaming  systems  should  support  these 
standards,  but  enforce  the  tag  model  found  on  many  Web  2.0  properties.  Tags  allow  individuals 
who  interact  with  content  to  describe  it  in  natural  language  terms  that  can  be  aggregated, 
searched,  and  filtered.  High-fidelity  e-learning  systems  should  allow  for  content  tagging  by  con¬ 
tent  owners  and  by  students.  In  turn,  searching  and  filtering  content  should  take  into  account  tags 
from  content  owners  and  students.  These  systems  should  also  take  advantage  of  natural  metadata, 
such  as  publication  date,  update  date,  content  type  and  creator  that  can  be  managed  by  the  system 
rather  than  by  the  content  owner. 

2. 6. 2. 2  Embedding 

High-fidelity  e-learning  systems  should  support  open  use  and  appropriate  distribution  of  learning 
material.  They  should  support  embedding  of  material  in  other  systems  such  as  websites,  blogs, 
and  wikis.  All  content  in  a  high-fidelity  e-learning  environment  should  have  a  direct  link  and, 
ideally,  a  code  snippet  to  allow  users  to  mash  up  the  learning  material  with  other  systems.  This 
must  be  balanced  against  permissions  on  the  content  and  licensing  or  revenue  model  in  place  for 
the  content. 

2.7  Progress  Tracking 

The  ability  of  a  course  provider  to  track  a  course  attendee’s  progress  is  necessary  to  enable  accu¬ 
rate  accounting  for  student  accomplishment  and  completion.  For  anytime,  anywhere,  and  at-any- 
pace  courses,  tracking  also  provides  a  valuable  place-marker  for  students  who  access  the  materials 
in  a  given  course  intermittently  over  an  extended  period  of  time. 

The  fidelity  of  progress  tracking  is  important.  The  learning  system  should  monitor  time  spent  with 
the  learning  materials  to  help  distinguish  students  who  are  fast-forwarding  through  the  materials 
from  those  who  are  spending  the  time  to  watch,  read,  listen,  and  leam.  Monitoring  interactivity  is 
also  valuable  to  help  understand  whether  the  student  is  actually  present  while  the  materials  are 
being  presented.  The  value  of  formative  assessments,  exercises,  and  labs  has  already  been  dis¬ 
cussed;  these  interactive  features  offer  the  added  benefit  of  enabling  the  system  to  better  track 
student  completion. 


3  http://ltsc.ieee.org/wg12/ 

4  http://dublincore.org/ 
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Affirmative  progress  tracking  is  particularly  important  for  training  that  is  provided  to  support 
compliance  or  regulatory  requirements.  To  support  such  uses,  the  learning  system  should  generate 
auditable  records  of  student  completion. 

When  courses  are  being  served  to  large,  diffuse  populations,  progress  tracking  can  help  the  course 
provider  understand  concurrency  and  evaluate  the  viability  of  peer-student  interactivity  in  the 
course. 

2.8  Balancing  Control 

Research  evidence  indicates  that  program  or  system  control  generally  yields  better  learning  results 
than  student  control  [Clark  &  Mayer  2003].  However,  users  accustomed  to  browsing  the  web  and 
using  digital  video  recording  devices  expect  a  high  level  of  control  when  interacting  through  a 
web  interface  or  watching  video  content.  Because  students  like  and  expect  control,  they  are  more 
likely  to  use  an  e-learning  course  if  some  amount  of  control  balance  has  been  implemented  [Clark 
&  Mayer  2003]. 

Chen  and  Liu  studied  three  types  of  student  control  in  a  web-based  course  and  correlated  usage 
patterns  and  control  preferences  with  cognitive  styles.  The  three  types  of  student  control  studied 
are  shown  in  Table  2.  These  are  typical  student  control  options  in  e-learning  systems  [Chen  &  Liu 
2008]. 

Table  2:  Three  Types  of  Student  Control  [Chen  &  Liu  2008] 


Control 

Purpose 

Tool 

Sequence 

Control 

To  allow  students  to  decide  the  se¬ 
quence  of  subjects  to  be  learned 

Hierarchicai  Map:  To  show  all  topics  and  subtopics  in  a 
hierarchical  way 

Alphabetical  Index:  To  list  keywords  in  alphabetical  order 
Back/Forward  Buttons:  To  see  the  page  previously  visited 

Content 

Control 

To  allow  students  to  control  the  se¬ 
lection  of  the  contents  they  wish  to 
learn 

Section  Button:  To  choose  from  three  sections  which  hold 
the  main  content 

Main  Menu:  To  present  the  main  topics 

Hypertext  Links:  To  connect  relevant  concepts 

Display  To  allow  students  to  choose  one  of  Display  Options:  To  include  an  overview,  examples,  de- 

Control  the  display  options  that  covers  the  tails,  and  references,  etc. 

same  concept 


Chen  and  Liu  found  that  students  with  different  cognitive  styles  showed  measureable  differences 
in  their  usage  patterns,  display  option  preferences,  and  navigation  preferences  in  the  web-based 
course  [Chen  &  Liu  2008],  Other  forms  of  student  control  provide  facilities  for  organizing  and 
annotating  materials  and  capturing  notes.  Vrasidas  advocates  student-centered  systems  in  which 
students  “organize  information  and  knowledge,  take  control  of  their  learning,  act  as  autonomous 
individuals  who  plan  and  execute  learning  tasks.  High-fidelity  e-leaming  systems  should  provide 
tools  that  allow  [students]  to  organize  information,  contribute  content,  and  engage  in  learning  ac¬ 
tivities”  [Vrasidas  2004], 

E-learning  systems  should  provide  course  designers  and  students  with  options  for  organizing,  in¬ 
dexing,  navigating,  and  annotating  the  learning  materials.  Doing  so  will  improve  the  likelihood 
that  students  use  the  learning  system  and  will  provide  the  flexibility  for  tailoring  to  various  cogni¬ 
tive  styles. 
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2.9  Using  the  Principles 


Organizations  implementing  technical  training  programs  can  use  these  principles  as  a  broad  set  of 
evaluation  criteria  for  selecting  or  implementing  a  training  delivery  system.  The  SEI's  VTE  has 
been  designed  to  reflect  and  implement  many  of  these  principles. 
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3  Virtual  Training  Environment  -  a  High-Fidelity  e-Learning 
System 


The  Software  Engineering  Institute’s  (SEI)  VTEis  a  high-fidelity  e-learning  system  that  has  been 
implemented  by  the  CERT  Program  at  the  SEI.  VTE  was  first  released  in  2005  and  has  continued 
to  evolve  and  mature.  VTE  provides  access  to  courses  for  thousands  of  users  and  offers  regularly 
added  and  updated  content. 

3.1  Background  and  History 

The  SEI  has  consistently  developed  and  delivered  distance  education  and  e-learning  programs. 
Shortly  after  September  11,  2001,  the  SEI  developed  an  m-depth  cyber  security  training  curricu¬ 
lum  in  support  of  the  Army  Reserve  Information  Operations  Command  (ARIOC).5  This  curricu¬ 
lum  consisted  of  two  five-day  training  courses  in  basic  and  advanced  principles  of  information 
assurance,  a  follow-on  course  in  cyber  forensics,  a  capstone  cyber  exercise,  and  an  assessment  to 
measure  mastery  of  the  material.  The  direct  instruction  was  heavily  weighted  towards  hands-on 
training.  Because  the  ARIOC  comprised  approximately  500  soldiers,  the  SEI  selected  a  train-the- 
trainer  model,  certifying  30  of  the  ARIOC’s  best  and  brightest  to  roll  out  the  training  program. 

The  SEI  provided  teaching  material  and  instructions  for  building  the  networks  needed  to  support 
the  hands-on  lab  environments  called  for  in  the  courses. 

Curriculum  development  and  training  were  completed  in  2004,  but  the  assessment  scores  did  not 
meet  expectations.  Feedback  from  the  ARIOC  indicated  that  as  reservists,  soldiers  did  not  have  an 
effective  means  to  review  the  material  prior  to  taking  the  assessment  exam.  ARIOC  instructors 
also  reported  great  difficulty  in  replicating  the  hands-on  lab  components  of  their  classes  due  to  the 
lack  of  hardware  and  software  as  well  as  constraints  imposed  by  local  security  policies. 

Taking  this  feedback  into  account,  the  SEI  initiated  a  project  to  address  these  issues.  The  objec¬ 
tives  of  the  project  were  as  follows: 

•  Allow  students  anywhere,  anytime  access  to  SEI  course  material  exactly  as  if  they  were  in 
class. 

•  Provide  on-demand  instructor  support  to  students  while  allowing  them  to  progress  at  their 
own  pace.  The  delivery  mechanism  should  be  instructor-facilitated,  not  instructor-led. 

•  Do  not  require  installation  of  any  software  on  student  machines. 

With  these  goals  in  mind,  the  SEI  built  an  application  and  converted  its  existing  library  of  infor¬ 
mation  security  training  material  to  meet  these  objectives.  Initial  development  began  in  November 
2004,  and  the  VTE  1.0  was  released  in  July  of  2005  with  on-demand  lecture  and  demonstration 
content  and  remotely-accessible  hands-on  labs,  all  available  through  a  standard  web  browser.  The 
SEI  released  an  upgrade  in  January  of  2006  and  a  complete  rewrite  of  the  application  (VTE  2.0) 
in  November  of  2006.  As  of  the  date  of  this  technical  report,  VTE  development  is  ongoing  and 
new  features  and  content  are  added  regularly. 

5  http://www.armyreserve.army.mil/USARC/OPS/USARRC/ARIOC/overview.htm 
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3.2  Implementing  the  Principles  -  VTE  Features  and  Benefits 

From  its  inception,  VTE  was  informed  by,  but  did  not  explicitly  address,  the  principles  of  high- 
fidelity  e-leaming  described  in  Section  2.  Over  the  lifespan  of  the  VTE,  the  SEI  has  made  an  ef¬ 
fort  to  leverage  these  principles  in  VTE 's  delivery  interface  and  content  authoring  processes.  This 
section  intro  duces  a  range  of  VTE  features  and  indicates  each  principle  they  support  in  brackets. 

3.2.1  Mimic  Classroom  Education  (Multimodality,  Personalization) 

Providing  an  experience  that  recreates  classroom  instruction  helps  to  set  a  learning  context  for  the 
student,  who  may  be  accessing  the  material  in  a  home  or  office  setting.VTE’s  lecture  player  inter¬ 
face  includes  the  same  elements  found  in  an  instructor-led  setting  such  as  slides,  visuals  of  the 
instructor  and  other  students,  recorded  demonstrations,  and  document  cameras.  These  elements 
reinforce  the  idea  that  the  student,  who  is  accessing  the  material  asynchronously,  is  actually  expe¬ 
riencing  the  class  and  can  benefit  from  questions  being  asked  by  other  students.  Figure  3  depicts  a 
typical  VTE  lecture  screen. 


Figures:  The  VTE  Lecture  interface 

3.2.2  Practice  with  Hands-on  Labs  (Lean-Forward  Learning) 

VTE  includes  a  robust  hands-on  tiaining  lab  capability  using  virtualization  technology  to  allow 
students  to  practice  concepts  asynchronously.  Course  and  content  owners  create  combinations  of 
virtualized  computers  in  a  sandboxed  network  and  publish  them  to  V  TE  along  with  instructions 
for  completing  the  lab.  Students  pm  vision  the  lab  and  access  it  remotely  through  a  web  browser, 
using  the  instructions  to  guide  their  progress.  This  allows  students  to  practice  concepts  using  ac¬ 
tual  systems  and  gain  practice  time  using  software  to  which  they  might  otherwise  not  have  access. 
After  the  students  complete  the  lab,  the  lab  resources  are  recycled  for  other  students.  All  of  the 
tools  used  in  the  labs  are  open  source. 
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3.2.3  Encourage  Annotations  and  Takeaways  (Personalization,  Lean-Forward 
Learnin  g) 

High-fidelity  e-leaming  is  an  example  of  Web  2.0  (or  as  TimB  emers-Lee  calls  it,  the  read/write 
web).fi  Stereotypically,  Web  1.0  sites  were  static  and  designed  to  present  infonnati on  through  one¬ 
way  communication  (e.  g.,  FAQs,  product  catalogs,  and  what' s  newpages).  Web  2.0  sites  are  de¬ 
signed  to  encourage  two-way  communication.  The  users  have  much  more  control  over  the  content 
of  a  Web  2.0  site.  Wikis,  blogs,  and  maskable  applications  such  as  Google  Maps  are  examples  of 
Web  2. 0  properties.  By  allowing  students  to  annotate  the  lecture  material  using  virtual  sticky 
notes  and  a  highlighter,  VTE  encourages  the  students'  creativity  and  allows  them  to  decide  how 
best  to  mark  the  material  for  future  reference.  Annotations  are  saved,  so  when  students  return  to 
the  lecture,  the  notes  theyVe  made  remain.  A  s  the  student  personalizes  the  material,  the  VTE  in¬ 
ter  face  begins  to  simulate  the  annotation  pro  cess  students  use  with  course  textbooks  and  class 
notes. 

3.2.4  Read  It,  Hear  It,  See  K,  Do  It,  Master  It  (M  ultimo  dal  ity) 

B  e cause  VTE  tiaining  material  is  accessed  asynchronously,  students  are  able  to  progress  at  their 
own  pace.  VTE  courses  are  typically  organized  as  sequences  of  related  material  referred  to  in  the 
system  as  a  topic.  Within  each  topic,  VTE  provides  written  material,  recorded  lectures,  narrated 
demonstrati  ons ,  and  hands-  on  training  lab  s .  Students  can  acces  s  any  item  as  many  time  s  a  s  they 
wish  until  they  achieved  their  learning  objectives.  Each  type  of  material  provides  the  student  with 
an  opportunity  to  more  deeply  engage  the  topic.  Figure  4  depicts  this  progression  of  content  rich¬ 
ness  and  topic  mastery. 


READ  IT 

HEAR  IT 

SEE  IT 

Documents 

Lecture  modules 

Recorded 

While  papers 

demonstrations 

Technical  briefs 

DO  IT 


Hands-on 
training  labs 


Figure  4:  The  VTE  Learning  "F’cwer  Curve" 


s  http:/yhev^.bbc.co.uk/1  technology/41 32752  .stm 
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3.2.5  Composing  Recorded  Lectures  (Modularity) 


Many  commercially  available  products  can  record  classroom  activity,  convert  it  to  a  web -based 
format,  and  immediately  publish  it  to  a  website.  With  these  products,  instant  gratification  and  ease 
of  use  come  at  a  cost  as  they  typically  offer  minimal  support  for  editing  the  material  once  it  is 
posted.  When  recording  content  for  VTE,  the  course  owner  and  an  SEI  instructional  designer 
collaborate  to  chunk  the  recorded  material  into  smaller  instruction  modules,  which  typically  last 
15  minutes.  These  individual  modules  are  published  and  then  re-assembled  into  the  original 
course.  This  extra  step  adds  time  and  complexity  to  the  publishing  process  but  significantly  im¬ 
proves  re-use  and  enhances  the  search  functionality. 

3.2.6  Accessing  Content  via  Training  and  Library  Modes  (Modularity) 

VTE  includes  two  complementary  approaches  for  accessing  content:  a  training  mode  designed  to 
guide  students  through  a  linear  sequence  of  learning  material,  and  a  library  mode  that  allows  stu¬ 
dents  to  scan/search  available  material  and  quickly  access  specific  modules  of  interest.  Training 
mode  supports  skill  building,  on-boarding  (bringing  new  staff  members  up  to  speed),  and  other 
learning  acquisition  models.  Students  use  library  mode  to  maintain  knowledge  or  quickly  refresh 
on  a  specific  topic. 

3.2.7  Using  VTE  Offline  (Accessibility,  Balancing  Control) 

Courses  in  VTE  can  be  published  to  a  local  file  system  and  copied  to  a  CD  or  DVD.  Students  can 
then  use  the  material  on  their  local  desktops  or  laptops  while  disconnected  from  the  internet.  This 
functionality  is  not  available  for  the  hands-on  labs  because  they  require  a  live  environment. 

3.2.8  Providing  a  Full  Transcript  (Multimodality,  Accessibility) 

The  SEI’s  content  publishing  process  ensures  that  each  piece  of  VTE  lecture  content  includes  a 
verbatim  text  transcript.  Transcripts  provide  the  following  benefits  for  VTE  users: 

•  The  ability  to  perform  deep  searches  (inside  and  across  material).  Students  can  use  keyword 
searching  to  quickly  find  material  related  to  a  narrow  topic  and  access  the  exact  portion  of  a 
lecture  that  referenced  that  key  word. 

•  The  ability  to  print,  download,  and  annotate  course  materials  to  improve  retention. 

•  Access  to  self-paced  learning.  Given  that  most  people  read  faster  than  the  instructor  speaks, 
students  can  move  through  the  material  at  their  own  pace  using  the  transcript  as  a  guide  to 
upcoming  material.  Though  by  default  the  transcript  follows  the  audio,  VTE  allows  the  stu¬ 
dent  to  unlink  the  transcript  and  treat  it  as  a  standalone  asset. 

•  Additional  time  to  review  materials.  In  some  cases,  students  may  not  natively  speak  the  in¬ 
structor’s  language.  Having  a  verbatim  transcript  allows  them  to  pause  the  audio  lecture  and 
look  up  words  they  need  to  clarify  without  missing  any  of  the  material. 

•  Compliance  with  American  with  Disabilities  Act  (ADA)  standards. 
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3.2.9  Content  Sequencing  and  Tracking  {Progress  Tracking} 


Like  many  learning  or  course  management  systems,  VTE  allows  content  owners  to  publish  ma¬ 
terial  in  sequence  and  to  track  student  progress.  Since  VTE  heavily  leverages  time  line -based,  syn¬ 
chronized  media,  it  is  possible  to  measure  the  time  a  student  spends  on  a  given  topic  and  in  VTE. 
VTE  includes  three  interaction  states  between  a  student  and  a  particular  piece  of  training  material: 
Not  Started ,  In  Process ,  and  Completed.  Each  time  a  slide  transition  occurs  while  a  student  is 
viewing  a  VTE  lecture,  the  server  records  the  event  and  a  timestamp.  Student  progress  is  marked 
In  Process  for  the  lecture  until  they  access  each  slide,  after  which  it  is  marked  Completed. 

When  students  have  completed  all  the  modules  in  a  course  (along  with  other  material  such  as  labs 
and  assessments),  they  receive  credit  for  completing  the  course.  VTE  students  can  generate  inte¬ 
rim  progress  summaries  and  print  completion  certificates  through  the  site.  Training  coordinators 
can  measure  the  progress  of  multiple  students  using  custom  reports. 

3.2.10  Virtual  Office  Hours,  Forums,  and  Breakout  Rooms  (Collaboration) 

To  provide  an  opportunity  for  live  ins  true  tor-to- student  interaction,  VTE  incorporates  virtual  of¬ 
fice  hours  when  instructors  are  available  for  personal  interaction.  In  a  university  setting,  office 
hours  are  a  means  for  students  to  obtain  additional  support  from  the  instructor  or  to  ask  questions 
that  are  not  relevant  to  the  rest  of  the  class.  When  establishing  a  class  in  VTE,  the  instructor  may 
post  a  list  of  specific  hours  when  he  or  she  will  be  available  in  the  virtual  meeting  room  and  in¬ 
clude  a  link  from  the  course  application  to  the  virtual  meeting  room.  Students  who  require  addi¬ 
tional  assistance  can  take  advantage  of  this  opportunity. 

VTE  allows  course  publishers  to  automatically  create  and  link  a  virtual  meeting  room  to  a  course. 
Students  and  instructors  may  jump  from  the  course  outline  directly  in  to  office  hours  without  re¬ 
quiring  a  separate  set  of  credentials.  While  in  the  virtual  meeting  room,  students  and  instructors 
can  share  desktops  and  applications,  publish  audio  and  video,  and  hold  text  chats. 

These  virtual  meeting  rooms  and  threaded  discussion  forums  allow  students  to  have  synchronous 
collaboration  with  one  another  and  with  instructors  to  further  support  in-depth  learning. 

3.3  VTE  Architecture 

The  VTE  architecture  consists  of  three  components: 

1 .  Core  Web  Application  and  Learning  Management  System.  The  web  application  layer  manag¬ 
es  the  user  interface  and  business  logic.  It  includes  a  learning  management  system  that  han¬ 
dles  course  creation,  enrollment,  progress  tracking,  and  reporting. 

2.  Video  Management  and  Delivery.  This  is  a  network  of  streaming  video  servers  and  cloud 
storage  used  to  improve  customer  experience  and  minimize  bandwidth  limitations.  This 
component  is  outsourced  to  a  commercial  content  delivery  network  provider. 

3.  Hands-on  Lab  System.  This  component  contains  a  lab  controller  and  scheduler,  a  storage  area 
network  for  managing  virtual  disk  images  (each  of  which  can  be  many  gigabytes),  and  several 
rackmount  servers  that  host  student  lab  environments. 
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Figure  5  illustrates  the  VTE  architecture  and  component  inter  feces. 
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Figure  5:  VTE  Conceptual  Architecture 

3.4  VTE  Use 

Since  its  launch  in  2004,  VTE  has  seen  steady  growth  and  adoption.  In  2008,  VTE  is  on  pace  to 
deliver  120,000  hours  of  training  material  to  over  1 1,000  active  users  across  the  U.S.  Government 
and  general  public.  Figure  6  shows  the  training  hours  delivered  by  VTE  per  quarter  for  the  past 
two  years. 
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Figure  6:  VIE  Training  Hours  Delivered  per  Quarter 

Neatly  20 ,000  registered  users  have  accessed  VTE  tiaining  material  since  inception,  and  over 
100,000  unique  unregistered  users  have  accessed  the  public  content  in  V  TE  that  does  not  require 
registration.  Figure  7  shows  the  total  VTE  registrations  per  quarter  for  the  past  two  years,  and 
Figure  8  shows  the  number  of  unique  registered  and  unregistered  users  that  have  visited  VTE  over 
the  past  two  years. 
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Figure  7:  VTE  registrations  per  quarter  for  past  two  years 
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Figure  8:  Unique  l/j'sjfs  to  I VTE  ty  Month  for  the  Pest  T wo  Years 

As  of  August  2003  there  are  740  unique  hours  of  lecture,  demonstration  and  hands-on  lab  training 
available  in  VTE.  This  material  has  been  combined  into  over  fifty  training  courses. 

Appendix  A  provides  a  detailed  case  study  and  ROI  financial  analysis  for  the  DoD  and  D  ISA’s 
use  of  VTE  for  information  security  certification  training.  Appendix  B  provides  a  summary  of 
VTE  use  by  the  U.S.  Secret  Service  for  computer  forensics  training. 

3.4.1  Comparisons  with  Direct  Instruction 

Since  VTE  provides  on-demand  access  to  recordings  of  direct  instruction,  it  is  possible  to  com¬ 
pare  the  volume  of  material  delivered  via  VTE  with  direct  instruction.  Using  a  ratio  of  one  day  of 
classroom  training  equaling  six  hours  of  online  instruction  (removing  lunch  and  breaks),  the 
120,000  hours  of  instruction  delivered  via  VTE  in  2003  is  equal  to  20,000  person-days  of  class¬ 
room  training. 

3.4.2  Scale 

The  two  most  challenging  components  of  VTE  to  accommodate  a  large  student  demand  are  the 
video  delivery  and  the  hands-on  labs.  VTE  is  architected  to  scale  by  adding  hardware  and  soft¬ 
ware  capacily  to  each  of  these  components.  Because  the  video  delivery  is  outsourced  to  a  com¬ 
mercial  content  delivery  network  (CDN),  VTE  can  scale  by  purchasing  additional  delivery  capaci¬ 
ty  to  the  theoretical  limits  of  the  CDN.  Labs  can  be  scaled  by  purchasing,  configuring  and 
installing  an  additional  server  for  delivery.  The  lab  controller  can  handle  thousands  of  delivery 
servers,  and  each  server  can  support  four  to  eight  concurrent  lab  environments  depending  on  size 
and  complexity  of  the  lab  environments  it  holds.  Review  of  VTE  usage  logs  shows  that  each  lab 
server  can  support  up  to  1,000  students  as  long  as  they  access  labs  asynchronously  (and  thus  have 
a  relatively  normal  distribution  of  usage). 
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3.5  Producing  VTE  Content 


The  two  most  typical  scenarios  for  producing  new  VTE  content  are  publishing  a  recorded  class 
and  creating  a  hands-on  lab.  As  with  any  learning,  education,  and  training  initiative,  getting 
started  with  VTE  begins  with  learning  objectives.  Typically  the  process  starts  with  an  existing  set 
of  course  resources  such  as  slides,  handouts,  and  qualified  instructors.  VTE  can  also  be  used  to 
publish  a  simple  demonstration  on  a  specific  topic. 

Note  that  this  section  focuses  on  a  content  producer  or  author  rather  than  an  instructor.  One  of  the 
design  goals  for  VTE  is  that  it  not  interfere  with  the  instructor’s  normal  delivery  and  student  inte¬ 
raction. 

3.5.1  Publishing  a  Recorded  Class 

This  section  describes  requirements  and  resource  estimates  for  including  a  recorded  class  as  part 
of  VTE. 

3.5.1 .1  Content  Requirements 

In  this  scenario,  an  organization  has  a  set  of  materials  that  represents  a  block  of  instruction,  where 
the  organization  typically  presents  the  material  in  a  live  classroom  setting.  The  instruction  is  sup¬ 
ported  with  slides,  usually  in  Microsoft  PowerPoint  format. 

3.5.1 .2  Infrastructure  and  Environment  Requirements 

Once  the  course  instructor,  slides,  materials,  students,  and  time  and  place  are  ready  to  go,  the  VTE 
producer  prepares  to  record  the  live  instruction.  At  a  minimum,  both  the  slides  and  the  instructor 
should  be  recorded.  It  is  also  helpful  to  record  the  students  or  to  capture  a  wide  shot  of  a  live 
classroom  experience  for  the  eventual  VTE  user. 

Periodically  transitioning  between  camera  views  can  make  the  material  more  visually  interesting. 
The  camera  directed  on  the  slides  may  not  be  a  camera  at  all  -  it  could  be  a  PC  add-in  card  de¬ 
signed  to  record  slide  images  from  VGA  output  .  Regardless  of  how  many  cameras  are  used,  they 
must  record  in  sync  with  one  another,  from  the  beginning  to  the  end.  In  particular,  the  timing  of 
slide  transitions  is  critical  to  ensure  that  the  online  student  experience  replicates  the  live  class  ex¬ 
perience. 

Footage  from  all  cameras  is  fed  into  a  non-linear  editor  (NLE)  for  post  production.  In  the  NLE,  a 
videographer  can  remove  extraneous  footage,  correct  color  or  audio  deficiencies  in  the  raw  ma¬ 
terial,  and  convert  recordings  to  a  format  suitable  for  online  display.  VTE  uses  Adobe  Flash  video 
as  the  player,  because  it  is  operating  system  agnostic  and  enjoys  a  very  high  market  penetration 
among  web  users.  As  a  result,  a  student  is  unlikely  to  need  to  download  and  install  new  software 
to  access  the  material. 

VTE  instructor  video  is  presented  at  a  resolution  of  320  x  240,  with  150  kilobits  per  second  (kbps) 
of  video.  The  slides  are  converted  to  vector  (Adobe  Flash)  format  and  are  presented  at  a  resolution 
of  640  x  480.  They  are  downloaded  to  the  student's  computer  when  the  student  accesses  a  lecture. 

As  part  of  the  NLE  process,  the  videographer  produces  an  audio  version  of  the  material  and  sub¬ 
mits  that  to  a  transcription  service  provider.  Typical  turnaround  is  48  to  72  hours.  The  transcript  is 
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time-coded  to  the  recorded  audio  file.  The  transcript  arrives  in  an  XML  format  that  can  be  easily 
converted  to  structured  data  for  use  in  VTE.  The  structured  transcript  data  is  merged  with  the  slide 
timing  data  to  create  a  master  event  timeline.  This  file  coordinates  the  events  within  the  VTE  on- 
demand  lecture  and  allows  the  student  to  move  through  the  material  as  they  see  fit. 

3.5.1 .3  Time  and  Schedule  Estimates  for  a  Typical  Course 

Based  on  the  SEI’s  experience  with  capturing  and  publishing  800  hours  of  training,  it  takes  one 
week  of  post-production  and  publishing  for  each  day  of  recorded  material  to  produce  a  quality 
result  from  a  captured  course.  So,  if  the  organization  captures  a  three-day  training  course,  it  can 
expect  to  expend  three  weeks  to  launch  the  course  on  VTE. 

3.5.2  Creating  a  Hands-On  Lab 

This  section  describes  requirements  and  resource  estimates  for  including  a  hands-on  lab  as  part  of 
VTE. 

3.5.2. 1  Content  Requirements 

A  hands-on  lab  should  reinforce  a  particular  learning  objective.  It’s  most  helpful  to  think  of  a  lab 
as  representing  a  change  in  state  -  of  a  network,  of  a  system,  of  a  piece  of  software,  etc.  The  stu¬ 
dent’  s  task  in  the  lab  is  to  change  the  environment  from  the  starting  state  to  the  finished  state,  so 
the  instructor’s  job  is  to  guide  the  student  through  that  process.  The  instructor  typically  accom¬ 
plishes  this  goal  by  creating  a  lab  document  that  describes  the  actions  the  student  must  undertake 
to  complete  the  exercise,  in  as  much  or  as  little  detail  necessary  to  meet  the  learning  objective. 

3. 5. 2. 2  Infrastructure  and  Environment  Requirements 

The  starting  point  for  creating  a  hands-on  lab  environment  is  one  or  more  virtual  disk  images, 
ideally  in  VMWare  format.7  This  product  allows  authors  to  create  virtual  machines  that  behave 
exactly  like  Intel-based  systems  -  Windows,  Linux,  Solaris,  etc. 

Depending  on  the  learning  objectives  for  the  lab,  it  may  be  possible  to  use  preconfigured  images 
from  VMWare’s  Virtual  Appliance  Marketplace.8  Many  of  the  images  are  free  for  use  in  training. 
Authors  should  be  aware  of  software  licensing  restrictions  because  VTE  can  create  hundreds  of 
instances  of  a  lab  environment  once  it  is  built,  which  may  violate  license  agreements. 

After  the  authors  have  established  the  authoring  environment  and  disk  images,  they  must  log  in  to 
the  virtual  machine(s)  and  install  any  custom  software  needed  for  the  scenario.  Once  the  starting 
state  of  the  lab  is  set,  the  author  should  take  a  snapshot  of  the  environment.  A  snapshot  is  a  special 
function  of  virtual  environments  that  creates  a  baseline  for  each  individual  image.  This  snapshot 
represents  the  starting  point  for  the  lab  exercise.  The  instructor  can  then  perform  the  exercise  to 
determine  whether  the  lab  meets  training  objectives.  During  this  first  dry  run  of  the  lab,  the  in¬ 
structor  can  take  screenshots  and  create  student  documentation  on  how  to  perform  the  lab  exer- 


Lab  authors  can  obtain  a  copy  of  the  free  VMWare  Server  product  from 
http  ://www.  vm wa  re .  com/pr  odu  cts/se  rver/. 

http  ://www.  vm wa  re .  com/ap  pi  i  an  c  es/ 
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cise.  When  the  instructor  is  satisfied,  the  snapshot  can  be  discarded.  The  base  images  are  ready  to 
be  loaded  into  VTE  where  a  new  lab  is  created  to  load  the  images  on  demand. 

3. 5. 2. 3  Time  and  Schedule  Estimates  for  a  Typical  Lab 

While  publishing  a  VTE  hands-on  lab  takes  only  minutes,  creating  the  lab  can  be  challenging  for 
a  novice  author  -  planning  and  conceiving  the  scenario  takes  a  lot  of  time  but  the  payoff  is  high. 
And,  given  that  the  lab  environment  can  be  replayed  as  often  as  needed,  scenario  reuse  is  high. 
The  SETs  experience  indicates  that  it  takes  about  one  staff-month  of  effort  to  produce  a  quality 
hands-on  training  lab  for  VTE. 
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4  Conclusion -The  Future  of  VTE 


The  SEI's  VTE  has  been  developed  to  reflect  the  principles  of  high-fidelity  e-learning.  It  offers 
the  user  a  web-based,  multimedia,  interactive  training  experience  that  can  be  accessed  anytime, 
anywhere.  This  environment  uses  state-of-the-art  software  and  hardware  to  train  users  with  tech¬ 
nical  and  managerial  roles  on  a  wide  range  of  information  assurance  topics. 

The  VTE  learning  model  presents  a  read  it,  hear  it,  see  it,  do  it,  master  it  progression  of  course 
lectures,  demonstrations,  and  hands-on  labs  to  meet  student  learning  objectives. 

The  SEI  is  continually  enhancing  the  VTE  platform  to  further  support  and  promote  the  principles 
described  in  this  report,  and  to  move  more  training  content  to  an  anywhere,  anytime  model.  The 
following  are  planned  enhancements  to  VTE  and  the  principles  they  support. 

•  The  Social  Learning  Management  System  [Personalization,  Collaboration,  Student 
Control,  Progress  Tracking] 

VTE’s  learning  management  system  (LMS),  which  includes  enrollment  in  courses, 
progress  tracking,  reporting,  and  instructor  support,  will  shift  from  a  centralized  model 
to  one  based  on  the  decentralized  model  made  popular  by  social  networking  sites  such  as 
MySpace,  Facebook  and  Linkedln.  This  shift  is  based  on  the  principle  that  users  and  or¬ 
ganizations  are  the  best  judges  of  their  training  needs  and  no  centralized  administrative 
office  is  able  to  react  quickly  enough  to  support  those  needs.  VTE’s  Social  LMS  will 
empower  users  and  organizations  to  build  a  learning  structure  that  mirrors  the  real  world. 
Instructors  and  students  will  be  able  to  easily  create  questions,  assemble  them  into  as¬ 
sessments,  rate  and  discuss  them,  and  export  them  to  other  systems.  Any  user  will  be 
able  to  create  courses  using  combinations  of  VTE  content  and  external  learning  re¬ 
sources  such  as  Wikipedia  pages  or  YouTube  videos.  Any  registered  user  will  be  able  to 
create  a  course;  any  other  user  can  request  to  be  enrolled  in  that  course.  Users  will  be 
able  to  build  relationships  with  other  users  and  support  each  others’  training  progress. 
VTE  will  also  include  built-in  functions  to  allow  users  to  set  goals  for  course  completion 
and  periodically  remind  and  encourage  users  to  complete  courses  on  a  schedule 

•  Formative  Assessment  via  hands-on  labs  [Lean-forward  Learning] 

The  SEI  will  expand  the  hands-on  lab  capability  to  allow  instructors  to  assign  labs  to 
students.  Lab  assignments  can  have  completion  dates  and  student  submissions  can  be 
graded  manually  by  the  instructor  or  programmatically  by  the  system.  These  formative 
hands-on  assessments  can  be  embedded  in  any  VTE  course  and  used  to  gauge  progress 
or  permit  the  student  to  enroll  in  other  courses. 

4.1  Whom  to  Contact;  Howto  Proceed 

Organizations  that  wish  to  discuss  leveraging  high-fidelity  e-learning  in  their  environment  should 
contact  VTE  Support  at  vte-support@cert.org  to  discuss  training  initiatives  and  next  steps. 
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Appendix  A  Defense  Information  Systems  Agency  (DISA) 
Case  Study 


Overview 

This  appendix  describes  the  adoption  of  VIE  by  the  DoD  through  a  sponsorship  from  the  DISA. 
This  appendix  covers  the  first  year  (FY2007)  of  the  DISA  sponsorship. 

The  data  presented  in  this  Appendix  focuses  on  cost  comparisons  and  return  on  investment.  The 
Appendix  acknowledges  the  need  for  data  that  indicates  the  quality  of  the  results  achieved 
through  VTE  training  such  as  student  course  completion  rates  or  student  certification  pass  rates. 
While  the  VTE  development  team  has  explored  several  possibilities  for  collecting  measures  of 
this  type,  no  such  data  is  currently  available.  For  future  research,  we  encourage  collecting  data  for 
measuring  VTE  training  effectiveness  and  will  continue  to  explore  the  means  to  do  so. 

In  December  2005,  the  DoD  issued  Directive  8570.1  Information  Assurance  Workforce  Improve¬ 
ment  Program  [DoD  2005],  In  summary,  the  directive  mandates  that  all  DoD  personnel  who  hold 
elevated  privileges  on  a  DoD  system  or  network  must  obtain  a  commercial  certification  within  six 
months  of  obtaining  the  privilege.  The  directive  identifies  two  different  types  of  positions  into 
which  individuals  are  to  be  categorized:  technical  (IAT)  and  managerial  (IAM).  Within  each  type, 
there  are  three  levels  that  describe  the  breadth  of  the  individual's  responsibility,  forming  a  matrix 
of  six  categories  of  personnel.  The  directive  and  subsequent  communications  from  the  DoD  map 
these  categories  to  commercially  available  certifications  in  information  security. 

The  directive  was  issued  without  specific  funding  guidance.  It  was  the  intent  of  the  DoD  that  ex¬ 
isting  training  budgets  for  the  information  assurance  (IA)  workforce  be  used  to  assist  staff  in  pre¬ 
paring  to  sit  for  a  certification  that  fits  their  job  classification.  The  DoD  IA  community  found, 
however,  that  the  cost  of  training  staff  to  meet  this  directive,  in  addition  to  the  rest  of  their  duties, 
significantly  strained  their  operating  budgets. 

Reacting  to  this  pressure,  the  DISA  contracted  with  the  SEI  in  November  2006  to  develop  prepa¬ 
ratory  courses  for  two  of  the  most  popular  commercial  certifications  that  met  the  8570  require¬ 
ments  (Security+  and  CISSP  [Certified  Information  Systems  Security  Professional]).  The  SEI  was 
to  make  these  courses  available  (along  with  other  supplemental  material)  to  a  portion  of  the  DoD 
IA  community  via  VTE.  DoD  personnel  who  were  required  to  obtain  a  certification  per  the  direc¬ 
tive  were  able  to  access  VTE  course  material  to  help  prepare  for  their  certification  test. 

In  the  first  year  of  the  program,  8,000  DoD  personnel  obtained  VTE  accounts  for  use  in  enhanc¬ 
ing  their  skills.  In  that  year,  users  accessed  over  38,000  hours  of  training  via  VTE.  The  program 
was  renewed  for  FY2008  and  now  averages  over  10,000  hours  per  month,  which  is  the  equivalent 
of  840  person-days  of  classroom  instruction  over  the  year.  According  to  George  Bieber,  Director, 
IA  Workforce  Improvement  Program,  "The  SEEVTE  support  to  the  DoD  has  had  a  significant 
positive  impact  on  DoD  8570  policy  implementation  by  providing  an  effective  means  of  preparing 
personnel  for  certification  testing.  Training  to  support  CISSP  and  Security+  training  have  reduced 
failure  rates  on  these  certifications  to  levels  below  those  experienced  by  individuals  in  the  private 
sector." 
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Financial  analysis  and  ROI  of  DISA  contract  data  for  2007 

Under  the  DISA  work  plan,  VTE  was  used  to  deliver  more  than  38,000  hours  of  training  in  the 
first  10  months  of  calendar  2007.  The  average  cost  of  a  delivered  training  hour  was  $22.49  in  that 
time  period.9 

To  analyze  and  better  understand  the  VTE  financial  performance  under  the  DISA  contract,  the 
performance  and  cost  data  from  the  DISA  contract  have  been  compared  to  annual  survey  data 
from  the  American  Society  of  Training  and  Development  (ASTD). 

THE  ASTD  REPORT 

ASTD  publishes  an  annual  State  of  the  Industry  report  based  on  a  survey  of  its  members  [ASTD 
2007].  The  report  and  the  survey  are  focused  on  workforce  training  through  workplace  learning 
and  performance  management  initiatives,  which  makes  the  data  well  suited  for  comparison  with 
VTE.  The  report  contains  detailed  statistics  about  training  costs  reported  by  the  members  and 
trends  in  workforce  training;  the  2007  report  contains  data  from  2001  through  2006.  One  trend 
discussed  in  the  report  is  technology-based  training  methods: 

“Persistent  usage  of  techno  logy -based  methods  has  dramatically  shaped  the  field.  As  organi¬ 
zations  deploy  e-learning  more  frequently,  the  use  of  instructor-led  learning  has  diminished. 
Discrete  learning  events  in  traditional  classroom  settings  are  gradually  shifting  to  learning 
experiences  that  are  occurring  at  the  workstation  and  at  the  pace  of  the  worker.  Many  organ¬ 
izations  have  adopted  e-leaming  platforms  in  recent  years  and  are  starting  to  see  their  in¬ 
vestments  bear  fruit.  Some  positive  trends  from  the  2006  data  include  an  increased  ratio  of 
learning  hours  used  to  learning  hours  made  available  (also  referred  to  as  reuse  ratio)  and  a 
greater  ratio  of  employees  to  WLP  [workplace  learning  and  performance]  staff  members. 
These  types  of  efficiency  gains  are  often  facilitated  by  techno  logy -based  solutions. 

The  downside  of  e-learning  sometimes  manifests  in  significant  development  costs.  Transi¬ 
tioning  to  an  infrastructure  that  supports  e-learning  typically  requires  a  substantial  invest¬ 
ment...”  [ASTD  2007] 

The  financial  data  provided  in  the  ASTD  report  do  not  separate  costs  for  conventional  training 
from  costs  for  techno  logy -based  training.  VTE  is  a  technology-based  training  solution,  so  the 
VTE  data  only  represents  techno  logy-based  training.  Any  comparisons  between  the  VTE  data  and 
the  ASTD  data  must  consider  this  important  difference.  ASTD  does  report  that  their  survey  indi¬ 
cates  that  30.28  percent  of  the  learning  hours  available  in  2006  were  in  the  form  of  technology- 
based  learning.  Within  the  ASTD  technology-based  learning  approximately  75  percent  was  avail¬ 
able  online  and  80  percent  of  the  online  content  was  self-paced.  Therefore,  for  2006,  approximate¬ 
ly  18  percent10  of  the  training  activity  reported  by  the  ASTD  survey  responses  in  2006  was  similar 
to  what  is  offered  by  VTE  (technology-based,  online,  and  self-paced). 

The  base  data  collected  by  the  ASTD  in  its  annual  survey  include  the  following: 


9  Calculated  by  dividing  the  total  contract  cost  for  1/1/07  through  10/31/07  by  the  total  number  of  hours  of  training 
delivered:  $858,250  -s-  38,157  training  hours  =  $22. 49/training  hour. 

10  30.28%  x  75%  x  80%  =  1 8% 
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•  Learning  Hours  Available  -  a  one-time  count  of  the  total  number  of  hours  of  learning  con¬ 
tent  available  in  all  forms  of  training  (classes,  workshops,  online,  etc.). 

•  Learning  Hours  Used  -  the  total  number  of  learning  hours  accessed  or  completed,  which  is 
calculated  by  multiplying  the  number  of  people  who  participate  in  a  learning  activity  by  the 
duration  of  that  learning  activity.  For  example,  if  a  four-hour  seminar  were  provided  and  1 00 
people  attended,  the  Learning  Hours  Used  would  equal  400. 

•  Direct  Learning  Investment  -  direct  expenditures  for  learning  activities.  The  survey  col¬ 
lects  this  data  across  a  number  of  categories,  but  does  not  distinguish  e-learning  investments 
from  other  learning  investments. 

From  these  baseline  measures  collected  in  the  survey,  the  ASTD  calculates  several  measures  that 
are  provided  in  the  report.  For  the  purposes  of  comparing  ASTD  data  with  VTE  data,  we  will  in¬ 
clude  the  following  measures: 

•  Cost  per  Learning  Hour  Available  -  the  report  does  not  indicate  how  this  measure  is  cal¬ 
culated,  but  based  on  the  survey  data  collected,  we  assume  that  it  is  Learning  Hours  Availa¬ 
ble  ^  Direct  Learning  Investment. 

•  Cost  per  Learning  Hour  Used  -  the  report  does  not  indicate  how  this  measure  is  calculated, 
but  based  on  the  survey  data  collected,  we  assume  that  it  is  Learning  Hours  Used  ^  Direct 
Learning  Investment 

•  Reuse  Ratio  -  the  ratio  of  Learning  Hours  Used  to  Learning  Hours  Available. 

The  ASTD  report  provides  the  data  consolidated  for  all  survey  responses,  and  also  provides 
breakdowns  by  various  member  types.  For  the  purpose  of  our  VTE  comparison,  we  use  the  con¬ 
solidated  data  since  there  are  no  category  data  that  are  a  close  fit  for  the  target  audience  of  VTE. 


Table  3  presents  data  from  the  ASTD  report  for  VTE  comparison. 
Table  3:  Data  for  Comparison  from  ASTD  Report 


Survey  Year 

2001 

2002 

2003 

2004 

2005* 

2006 

Cost  per  Learning 
Hour  Available 

$1,109.28 

$924.32 

na 

$1,543.28 

Cost  per  Learning 
Hour  Used 

$36.79 

$45.72 

$62.89 

$49.99 

na 

$54.25 

Reuse  Ratio 

29.00 

36.07 

na 

41.31 

* 2005  data  is  not  provided  by  the  ASTD  due  to  a  change  in  the  survey  process 

VTE  DATA  FOR  DISA  SUPPORT 


An  agreement  was  established  between  the  SEI  and  the  DISA  in  November  2006  to  provide  VTE- 
based  training  to  8,000  eligible  training  participants. 


The  VTE  system  automatically  logs  usage  data.  Data  from  VTE  and  from  the  DISA  cost  history 
have  been  collected  to  calculate  measures  for  comparison  to  the  ASTD  study  as  shown  in  Table  4: 


Table  4:  VTE  Data  for  Comparison 


2007 


Survey  Year 


Cost  per  Learning  $2,657.12 


28  |  CM  U/S  E  l-T  R- 2009-005 


Hour  Available 


Cost  per  Learning 

$22.49 

Hour  Used 

Reuse  Ratio  118 


»  Cost  per  Learning  Hour  Available  was  calculated  by  dividing  the  total  number  of  VTE 

content  hours  available  to  DISA  participants  by  the  total  cost  of  the  D  ISA  agreement  for  the 
10-month  period  (January  1,  2007  -  October  3 1, 2007). 

»  Costper  Learning  Hour  Used  was  calculated  by  dividing  the  total  number  of  VTE  content 

hours  accessed  by  D  ISA  participants  by  the  total  cost  of  the  DISA  agreement  for  the  1 0- 
monlh  period  (January  1,  2007  -  October  31,  2007). 

»  Reuse  Ratio  was  calculated  by  dividing  the  total  number  of  VTE  content  hours  accessed  by 

DISA  participants  by  the  total  number  of  VTE  content  hours  available  to  DISA  participants 
for  the  10-month  period  (January  1,  2007  -  October  31,  2007). 

COMPARISON  OF  VTE  DATA  TO  ASTD  DATA 

The  following  charts  provide  a  graphical  comparison  of  VTE  and  ASTD  measures.  Paragraphs 

following  each  chart  pm  vide  interpretation  of  the  comparisons. 


2001  2002  2003  2004  2005  2006  2007 


Figure  9:  Cost  per  Learning  Hour  Used 

The  Cost  per  Learning  Hour  Used  from  VTE  is  lower  than  in  the  ASTD  survey  results.  This 
should  be  anticipated  since  VTE  data  is  for  e-learning  only  while  ASTD  data  is  for  e-leaming  plus 
conventional  training  activities.  Since  most  of  the  costs  for  e-leaming  solutions  are  for  initial  set 
up  of  the  training,  it  should  be  expected  that  the  costper  learning  hour  used  decline  rapidly  as 
additional  people  access  the  available  training.  The  average  ASTD  Costper  Learning  Hour  Used 
for  2001-2006  is  $49.93,  whichis  more  than  double  the  cost  experienced  with  VTE  in  2007. 
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2001  2002  2003  2004  2005  2006  2007 


Figure  10:  Cost  per  Learning  Hour  Available 

Cost  per  Learning  Hour  Available  for  VIE  is  higher  than  shown  in  the  A  STD  survey  data.  This  is 
to  be  expected  given  that  VTE  is  purely  e-leaming  while  ASTD  data  contains  both  e-learning  and 
conventional  training.  In  analysis  of  its  own  results,  the  ASTD  indicates  that  a  higher  cost  per 
learning  hour  available  should  be  anticipated  when  funding  the  infrastructure  to  establish  e- 
leaming  solutions.  The  average  ASTD  C  ost  per  Learning  Hour  Available  for  2003-2006  is 
$1,192,  whichis  less  than  half  the  VTE  Costper  LeamingHour  for  2007. 

The  Reuse  Ratio  for  VTE  is  substantially  greater  lhan  shown  in  the  ASTD  data.  This  too  is  to  be 
expected  given  that  VTE  is  an  e-leaming  platform.  The  hours  available  from  an  e-leaming  solu¬ 
tion  are  accessible  on-demand,  subject  to  few  limitations  (such  as  bandwidth).  The  ASTD  average 
Reuse  Ratio  for  2003-2006  is  35,  whichis  about  one-third  of  the  VTE  Reuse  Ratio  for  2007. 

Return  on  Investment 

The  cost  data  provided  in  the  ASTD  report  can  be  used  to  value  the  training  hours  delivered  to 
DISA  through  VTE  and  calculate  a  return  on  investment 

VTE  was  used  to  deliver  38,1 57  hours  of  training  for  DISA  during  the  period  from  January  1 , 

2007  through  October  3 1 ,  2007.  ASTD  reports  that  the  average  cost  per  learning  hour  delivered 
by  its  members  in  2006  was  $54.25.  According  to  the  ASTD  data,  the  value  of  VTE -delivered 
training  is  therefore  $2,070,0 17. 11  The  total  cost  to  DISA  for  the  VTE- delivered  training  was 
$858,250.  This  represents  a  cost  savings  to  the  DISA  of  $1,21 1,767  as  compared  to  what  they 
could  have  expected  to  pay  at  prevailing  industry  average  costs.  The  total  return  on  investment  for 
the  DISA  is  Hlpercent.12 

Conclusions 

The  comparison  of  VTE  data  for  2007  to  ASTD  data  from  2001  through  2006  provides  evidence 
for  the  benefits  of  an  e-learning  solution  like  VTE.  Despite  what  is  clearly  higher  development 


11  $54.25  per  hour  X  38,157  hours  =  $2,070,017.25 

12  ($2 ,070, 017  -  $853 ,250)  /  $653,250  =  1 41  % 
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costs  for  a  given  hour  of  training,  the  VTE  solution  provides  substantially  lower  cost  per  training 
hour  delivered  and  a  much  higher  reuse  ratio  for  training  hours  than  is  currently  being  expe¬ 
rienced  in  blended  (e-learning  plus  traditional  training)  programs  for  workplace  training. 
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Appendix  B  U.  S.  Secret  Service  Experiences  with  VTE 


The  appendix  presents  the  executive  summary  from  the  Department  of  Homeland  Security  U.  S. 
Secret  Service  Criminal  Investigative  Division's  "Virtual  Training  Environment  (VTE)  Pilot 
Study"  Final  Report  (USSS  2005).  It  is  reprinted  with  permission. 

In  early  2005,  Camegie-Mellon  University,  Software  Engineering  Institute,  CERT,  launched 
an  initiative  to  create  an  on-line  or  “virtual  training  environment”  for  delivery  of  course  ma¬ 
terials  related  to  its  Knowledge  in  Depth  for  Defense  in  Depth  (KD3)  program. 

The  KD3  program  employs  a  Virtual  Training  Environment  (VTE)  system  to  deliver,  via  the 
internet,  program  content  in  an  always-available  manner.  VTE  . .  allows  students  to  train 
without  regard  to  schedule,  physical  location,  or  presence  of  instructor  in  an  anywhere,  any¬ 
time  environment13.” 

In  March,  informal  discussions  began  with  VTE  development  staff  about  the  potential  for  its 
use  by  the  Electronic  Crimes  Special  Agent  Program  (EC SAP)14  program  as  a  means  to  de¬ 
liver  core  and  specialized  training,  and  continuing  education  to  EC  SAP  personnel.  VTE  was 
described  as  distance  learning  tool  that,  in  addition  to  providing  material  via  video,  audio, 
and  text,  incorporated  ‘virtual  labs’  wherein  students  could  safely  and  repeatedly  practice  the 
skills  and  techniques  learned  during  instructional  portions  of  a  course. 

In  May,  Criminal  Investigative  Division  (CID)/Electronic  Crimes  Section  (ECS)  announced 
its  intention  to  research  “distance  learning”  for  the  EC  SAP  program.  VTE  was  offered  as  a 
possible  solution  and  a  Pilot  Study  was  proposed  to  assess  VTE’s  strengths,  weaknesses  and 
compatibility  with  ECS’s  vision  of  distance- learning. 

VTE  was  demonstrated  during  the  July,  2005  EC  SAP  conference15  in  San  Diego,  CA  and  25 
attendees  volunteered  for  the  Pilot  Study.  The  Study  took  place  over  a  two-month  period 
(Oct  1 st  thru  November  30th)  and  consisted  of  two  parts;  Operational  and  Assessment.  The 
operational  portion  began  October  1 st  and  concluded  at  the  end  of  the  month.  During  this  pe¬ 
riod  Study  participants  viewed  various  demos,  tutorials  and/or  courses  and  conducted  one  (or 
more)  associated  labs.  The  Assessment  portion  began  November  1 st  and  concluded  Novem¬ 
ber  30th.  During  this  time  participants  were  asked  to  record  their  experience(s)  using  VTE 
and  to  rate  its  features,  functionality  and  applicability  to  the  EC  SAP  program.  Participants 
recorded  their  observations  and  opinions  thru  either  on-line  or  printed  survey. 

Overall,  participants  rated  VTE’s  features  and  functionality  between  seven  (7)  and  ten  (10) 
on  a  1 0-point  scale.  Areas  rated  included  access,  navigation,  and  audio  and  video  content. 
Overall,  14  of  1616  respondents  answered  ‘yes’  when  asked  if  they  considered  VTE  of  bene¬ 
fit  to  the  EC  SAP  program  and  if  it  should  be  adopted  for  use  by  the  Secret  Service  for  the 
ECS  AP  program.  Moreover,  one  participant  reported  that  he  had  used  VTE  to  assist  him 
with  resolving  a  technical  roadblock  he  confronted  during  a  recent  digital  examination. 


13  Source:  KD3  Information  Sheet 

14  "Agents  assigned  to  ECSAP  are  computer  investigative  specialists,  qualified  to  conduct  examinations  on  many 
types  of  electronic  evidence,  including  computers,  personal  data  assistants,  telecommunications  devices,  elec¬ 
tronic  organizers  and  other  electronic  media."  [http://www.ustreas.gov/usss/criminal.shtml] 

15  Presentation  by  Mr.  Richard  Nolan,  VTE  Development  Team  Manager 

16  The  remaining  two  respondents  answered  ‘yes,  with  reservations’ 
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In  short,  CERT’s  Virtual  Training  Environment  is  a  high-performance,  professional,  any- 
time-any where  distance  learning  program.  It  is  capable  of  delivering  a  wide  variety  of  con¬ 
tent  (video,  audio,  text)  in  a  manner  which  is  both  intuitive  and  flexible.  What  makes  VTE 
unique,  however,  is  its  ability  to  let  students  practice  what  they  leam  in  a  safe  environment 
through  its  virtual  labs.  VTE,  by  design,  is  engineered  to  adapt  to  the  changing  needs  and  cri¬ 
teria  of  its  users. 
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